Visual search refers to the challenge of finding specific, relevant objects (also defined as targets) in a tactical scene. Imagery is commonly used as a visual search aid in tactical situations, such as military actions and security operations. For example, an image of a scene may be produced to analyze it and extract from it discernable features of an object of interest. When imaging tactical scenes in varying conditions (e.g., noise, clutter, dynamic range, and/or dynamic motion), perception of subtle anomalies and irregularities in such imagery can be difficult. This is especially true in imaging situations in which such conditions cannot be controlled in terms of signal to background, signal to noise, and signal to clutter.
Computer-aided techniques may be used to improve the ability to perceive a target in a scene and/or accelerate detection and/or identification of such a target, either with or without a man-in-the-loop. However, vast amounts of available imagery data can make it difficult to quickly and accurately find and extract relevant and/or actionable information in the field, and particularly when target detection results are required by an observer in near real-time.
Image enhancement techniques may be used to facilitate visual interpretation and understanding of digital imagery. Although a particular image may not be optimized for visual interpretation, an advantage of digital imagery is that digital pixel values in an image may be manipulated (i.e., corrected) to highlight various features present in the image that may be visually obscured. Because of the large variations in the spatial characteristics provided by a diverse range of tactical scenes (e.g., forests, deserts, grassy fields), no generic correction can optimally account for and display all targets in any given scene. Rather, for any particular image and/or tactical application, a custom correction may be necessary to detect a target. In some instances, a correction can be provided by one or more filters that when applied to a digital image can enhance or suppress various features contained therein. A filter represents a class of digital image correction functions that can help declutter complex visual scenes and speed the identification of targets embedded in the clutter.
Frequency filters are one class of filter that process an image in the Fourier domain. Frequency filters selectively filter the spatial frequencies in a particular image. The spatial frequencies of an image vary according to the content the image contains. An image containing small details and sharp edges, for example, contains higher spatial frequencies than images having content with lower resolution. A scene's spatial content interacts with the human visual system's own modulation transfer function (MTF). The inverse MTF is referred to as the contrast threshold function (CTF). When applying a frequency filter to an image, the image—which in its visually normal state is referred to as being in the spatial domain—is Fourier transformed, multiplied with a filter function in a pixel-by-pixel fashion, and then re-transformed into the spatial domain using an inverse Fourier Transform. All frequency filters can in theory be implemented in the spatial domain and, if there exists a simple kernel for the desired filter effect, filtering in the spatial domain is computationally economical to perform. Frequency filtering is more appropriate in those instances where a simple kernel cannot be found in the spatial domain.
In addition, an image can be filtered by orientation to aid in target detection (e.g., frequency filters may be extended to two dimensions to create filters which are selective for orientation). Orientation filtering may exploit the tendency of natural and man-made scenes to be characterized by vertical and horizontal content, which may contribute to the ‘oblique effect’ in the human visual system (i.e., greater sensitivity to the cardinal orientations, and a greater number of neurons tuned to these orientations).
The selection of the filter function(s) applied to an input image determines their effects. Three types of frequency filters, for example, can be applied to enhance an image: low pass, high pass, and band-pass. A low-pass filter attenuates high frequencies and maintains low frequencies unchanged. The result in the spatial domain is equivalent to that of a smoothing filter, as the blocked high frequencies correspond to sharp intensity changes (i.e., fine-scale details and noise in the spatial domain image). A high-pass filter provides edge enhancement or edge detection in the spatial domain because, as noted above, an image having edges contains many high frequencies. When a high-pass filter is applied, areas of rather constant gray level resolution, which consist of mainly low frequencies, are suppressed. A band-pass filter attenuates very low and very high frequencies but retains a middle range band of frequencies. Band-pass filtering can be used to enhance edges (suppressing low frequencies) while reducing the noise at the same time (attenuating high frequencies).
As described above, selectively filtering imagery has been shown to speed and guide human visual search of targets (a problem space referred to herein as tactical cuing). The following references, which are not admitted to be prior art with respect to the present invention by inclusion in this section, are incorporated by reference for background on the theoretical basis of selective filtering, the photos, filters, and study test stimuli illustrated in FIGS. 1-7:    (1) Pinkus, A. R., Garrett, J. S., Paul, T. M., & Pantle, A. J. (2015). Effects of the experimental manipulation of Fourier components of naturalistic imagery on search performance and eye-tracking behavior. Proc SPIE, Vol. 9474-34 S7;    (2) Pinkus, A. R., Poteet, M. J., & Pantle, A. J. (2008). Dynamic stimulus enhancement with Gabor-based filtered images. Proc SPIE, Orlando, Fla., Vol. 6968-63;    (3) Pinkus, A. R., Poteet, M. J., & Pantle, A. J. (2013). Search performance with discrete-cell stimulus arrays: filtered naturalistic images and probabilistic markers. Psychological Research, Vol. 77, 277-302;    (4) Dosher B A, Lu Z-L. (2000). Noise exclusion in spatial attention. Psychological Science, 11:139. doi: 10.1111/1467-9280.00229; and    (5) Smith A T, Singh K D, Greenlee M W. (2000). Attentional suppression of activity in the human visual cortex. Neuroreport, 11(2):271-277.
FIG. 1 shows a pre-filtered source image 100 depicting an exemplary aerial view of the Royal Dutch Army training village in Reek, The Netherlands. FIG. 2 depicts the different filters 200 constructed in the Fourier-domain as applied to modify the FIG. 1 source image 100. Illustrated are four filters: (1) narrow orientation 205, (2) broad orientation 215, (3) middle band-pass 225, (4) notch low- and high-pass 235. As illustrated, white areas pass spatial frequencies and black areas block spatial frequencies.
The narrow orientation filter 205 passes all spatial frequencies with orientations between 0-12.5 degrees. The broad orientation filter 215 passes all spatial frequencies with orientations between 0-67 degrees. The middle band-pass filter 225 passes only spatial frequencies from 3-8 cycles per segment (all orientations). The notch filter 235 blocks spatial frequencies 3-8 cycles per segment (all orientations).
FIG. 3 illustrates exemplary visual effects of applying the different filters of 200 of FIG. 2 to exemplary image 302 (Note: This smaller area 302 of source image 100 is selected and enlarged for clarity of illustration herein). Various filter combinations 305, 315, 325, 335 are shown as applied to (i.e., convolved with) segmented areas (patches) of the source image 302. As mentioned the theoretical references listed above, selectively filtering imagery has been shown to speed and guide human visual search of targets. Selective attention in visual search involves both stimulus (referred to herein as “target”) enhancement and suppression of competing (referred to herein as “distractor”) information. Applying appropriately constructed filters, as described herein, to a digital image can help declutter complex visual scenes and speed the identification of obscured targets embedded in clutter.
The exemplary target cuing display 400 of FIG. 4 shows the filtered image 302 anatomically segmented into seven (7) rows and seven (7) columns defining forty-nine (49) patches. As illustrated in FIG. 4, a filter of band-pass type 325 (from FIG. 3) may be applied to create each of the 49 patches, although other types of filters (including orientation-narrow type 305, orientation-broad type 315, and notch: low and high pass type 335) may be applied to produce alternative enhancement effects. For purposes of the illustrated example, a centrally-located patch 405 may be assumed to represent the search target and all the other (48) patches disposed about the search target 405 may be assumed to serve as distractors (or distractor segments). When the target 405 and distractor patches (clutter) are altered using the same filter (as shown, a filter of band-pass type 325), the visual search times and number of fixations increase. The tracking lines 410, 420, 430 in FIG. 4 illustrate exemplary eye-scanning behavior of the source image 302 with the same filter applied to both the target 405 and distractor segments. As shown, numerous fixations (i.e., direction-changing points along the search paths 410, 420, 430, defined herein as “hits”) are carried out to find the target pattern's location 402 within the target patch 405. Similarly, the exemplary target cuing display 500 of FIG. 5 shows heat mapping of the source image 302 with the same filter (as illustrated, a filter of band-pass type 325) applied to both target 505 and distractor segments. Similar to FIG. 4, multiple positive heat readings present as object “hits” in distractor segments may be expected to result in longer tracking times (e.g., multiple fixations) to find the target pattern's location 502 within the target patch 505.
Referring now to FIG. 6, and referring again to FIG. 3, a target area of an input image may be altered with one type of filter and distractor patches (clutter) may be altered using a different filter, advantageously resulting in fewer fixations and shorter search times to find a target pattern. For example, and without limitation, in the exemplary target cuing display 600 the centrally-located patch 605 in the filtered image 302 may define the search target whereas all the other (48) patches may serve as distractors (similar to prior art FIG. 4). The tracking lines 610, 620, 630 in FIG. 6 illustrate exemplary eye-scanning behavior of the source image 302 in which one filter (e.g., vertical line enhancement) may be applied to half of the segments including the target 605 and a different filter (e.g., horizontal line enhancement) may be applied to the other half of the segments (all distractors). As shown, typical eye-scanning behavior may exhibit fewer fixations carried out to find the target pattern's location 602 compared to the fixations experienced to find the target pattern's location 402 using the single-filter approach from prior art FIG. 4. Similarly, the exemplary target cuing display 700 of FIG. 7 shows heat mapping of the source image 302 in which one filter may be applied to half of the segments including the target 505 and a different filter may be applied to the other half of the segments (all distractors). As shown, fewer object “hits” in distractor segments may result in shorter times to find the target pattern's location 702 within the target segment 705 than were experienced using the single-filter approach from prior art FIG. 5.
A need exists for a target cuing solution that tailors the application of filters to both the target potential areas and the non-target areas of a tactical scene, such that fewer fixations and shorter search times to find a target pattern may be realized.
This background information is provided to reveal information believed by the applicant to be of possible relevance to the present invention. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art against the present invention.